[SPARK-37558][DOC] Improve spark sql cli document #34821

AngersZhuuuu · 2021-12-06T12:19:57Z

What changes were proposed in this pull request?

Current Spark SQL CLI doc just show simple usage. In this pr we add a detail doc for spark sql cli.

Why are the changes needed?

Make doc about Spark SQL CLI more clear. help user to use this command line tool.

Does this PR introduce any user-facing change?

No

How was this patch tested?

SparkQA · 2021-12-06T13:01:07Z

Test build #145954 has finished for PR 34821 at commit 01848dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-06T13:30:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50427/

SparkQA · 2021-12-06T14:21:34Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50427/

docs/sql-distributed-sql-engine.md

docs/sql-distributed-sql-engine-spark-sql-cli.md

SparkQA · 2021-12-07T03:03:55Z

Test build #145970 has finished for PR 34821 at commit 7a9ecbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-12-07T03:13:08Z

ping @cloud-fan

SparkQA · 2021-12-07T03:36:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50445/

SparkQA · 2021-12-07T03:59:59Z

Test build #145972 has finished for PR 34821 at commit a0218f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-12-07T04:31:45Z

docs/sql-distributed-sql-engine-spark-sql-cli.md

+## Spark SQL CLI Interactive Shell Commands
+
+When `./bin/spark-sql` is run without either the `-e` or `-f` option, it enters interactive shell mode.
+Use `;` (semicolon) to terminate commands, but user can escape `;` by `\\;`. Comments in scripts can be specified using the `--` prefix.


What if ; is not the end of the line? does it still terminate commands?

I think we should explain more about ;

what if it's inside a comment? SELECT /* xxx; */ 1;

is ; the only way to terminate commands? what if I type SELECT 1 and then press enter?

can we put multiple commands in a single line? SELECT 1; SELECT 2;

How about current?

SparkQA · 2021-12-07T04:32:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50447/

SparkQA · 2021-12-07T04:34:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50445/

SparkQA · 2021-12-07T05:32:50Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50447/

SparkQA · 2021-12-07T06:00:08Z

Test build #145977 has finished for PR 34821 at commit cea5be4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-12-07T06:10:08Z

docs/sql-distributed-sql-engine-spark-sql-cli.md

+Use `;` (semicolon) to terminate commands. Notice:
+1. CLI use `;` to terminate commands only when it's at the end of line and it's not escaped by `\\;`.
+2. `;` is the only way to terminate commands, if user type `SELECT 1` and press enter, console will just wait for input.
+3. If user type multiple commands in one line like `SELECT 1; SELECT 2;`, commands `SELECT 1` and `SELECT 2` will be executed separatly.


and CLI only shows the result of the last command?

and CLI only shows the result of the last command?

No, both result.

spark-sql> select 1; select 2; 1 Time taken: 3.819 seconds, Fetched 1 row(s) 2 Time taken: 0.077 seconds, Fetched 1 row(s)

If the user types multiple commands in one line like SELECT 1; SELECT 2;, then commands SELECT 1 and SELECT 2 will be executed separately.

cloud-fan · 2021-12-07T06:11:05Z

docs/sql-distributed-sql-engine-spark-sql-cli.md

+     /* This is a comment contains ;
+     */ SELECT 1;
+     ```
+  it will terminate commands into  `/* This is a comment contains ` and `*/ SELECT 1`.


and we hit failures when executing /* This is a comment contains

spark-sql> /* This is a comment contains ; Error in query: Unclosed bracketed comment(line 1, pos 0) == SQL == /* This is a comment contains ; ^^^ spark-sql> */ SELECT 1; Error in query: extraneous input '*/' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 5) == SQL == */ SELECT 1 -----^^^ spark-sql>

SparkQA · 2021-12-07T06:38:46Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50452/

SparkQA · 2021-12-07T07:16:23Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50452/

AngersZhuuuu · 2021-12-08T02:51:54Z

Any more suggestion?

docs/index.md

docs/sql-distributed-sql-engine-spark-sql-cli.md

srowen · 2021-12-08T03:23:47Z

docs/sql-distributed-sql-engine-spark-sql-cli.md

+   -- This is a comment;
+   SELECT 1;
+   ```
+   This comment line will just be ignored. If `;` in a bracketed command and not at the end of line,


then this comment line will be ignored. If ; appears in a bracketed command

(and similarly for the other lines below)

docs/sql-distributed-sql-engine-spark-sql-cli.md

srowen · 2021-12-08T03:25:07Z

docs/sql-distributed-sql-engine-spark-sql-cli.md

+</tr>
+<tr>
+  <td><code>dfs &lt;dfs command&gt;</code></td>
+  <td>Executes a <a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfs">dfs command</a> from the Hive shell.</td>


Is this only for executing commands vs HDFS or would it work on other distributed storage?
We should say HDFS if it's HDFS.
Also, not "Hive shell" here right?

Done, how about current?

docs/sql-distributed-sql-engine-spark-sql-cli.md

SparkQA · 2021-12-08T07:10:49Z

Test build #146000 has finished for PR 34821 at commit 36ce29f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-08T07:43:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50476/

Co-authored-by: Wenchen Fan <[email protected]>

AngersZhuuuu · 2021-12-14T02:48:10Z

Error in loadNamespace(x) : there is no package called ‘pkgdown’
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
Can you rebase this PR to pass Github Actions? It checks the documentation build.

Done

SparkQA · 2021-12-14T03:31:51Z

Test build #146158 has finished for PR 34821 at commit d9f150c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-14T04:01:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50631/

SparkQA · 2021-12-14T04:48:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50631/

docs/sql-distributed-sql-engine-spark-sql-cli.md

cloud-fan · 2021-12-14T15:27:39Z

docs/sql-distributed-sql-engine-spark-sql-cli.md

+## Path interpretation
+
+Spark SQL CLI supports running SQL from initialization script file(`-i`) or normal SQL file(`-f`), If path url don't have a schema component, the path will be handled as local file.
+For example: `/path/to/spark-sql-cli.sql` equals to `file:///path/to/spark-sql-cli.sql`.


let's also mention that people can use Hadoop supported filesystem such as s3://...

cloud-fan

LGTM except for a few minor comments

SparkQA · 2021-12-14T15:40:51Z

Test build #146189 has finished for PR 34821 at commit 453f260.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-14T16:15:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50662/

SparkQA · 2021-12-14T16:53:29Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50662/

Co-authored-by: Wenchen Fan <[email protected]>

SparkQA · 2021-12-15T06:48:06Z

Test build #146220 has finished for PR 34821 at commit d4f6778.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-15T07:18:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50694/

SparkQA · 2021-12-15T07:37:21Z

Test build #146223 has finished for PR 34821 at commit 2b6f130.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-12-15T07:45:32Z

LGTM except for a few minor comments

Scrrenshot updated in pr desc

SparkQA · 2021-12-15T08:06:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50697/

SparkQA · 2021-12-15T08:20:10Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50694/

SparkQA · 2021-12-15T09:07:33Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50697/

SparkQA · 2021-12-15T09:34:46Z

Test build #146227 has finished for PR 34821 at commit 147d8b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-15T10:13:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50701/

SparkQA · 2021-12-15T11:13:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50701/

cloud-fan · 2021-12-15T14:28:28Z

thanks, merging to master!

[SPARK-37558][DOC] Improve spark sql cli document

01848dc

github-actions bot added the DOCS label Dec 6, 2021

srowen reviewed Dec 6, 2021

View reviewed changes

update

7a9ecbf

Update sql-distributed-sql-engine-spark-sql-cli.md

a0218f8

AngersZhuuuu changed the title ~~[WIP][SPARK-37558][DOC] Improve spark sql cli document~~ [SPARK-37558][DOC] Improve spark sql cli document Dec 7, 2021

cloud-fan reviewed Dec 7, 2021

View reviewed changes

AngersZhuuuu added 2 commits December 7, 2021 13:11

Update sql-distributed-sql-engine-spark-sql-cli.md

1c59308

Update sql-distributed-sql-engine-spark-sql-cli.md

cea5be4

cloud-fan reviewed Dec 7, 2021

View reviewed changes

srowen requested changes Dec 8, 2021

View reviewed changes

AngersZhuuuu added 2 commits December 8, 2021 14:28

Merge branch 'master' into SPARK-37558

8eb04bf

follow comment

36ce29f

AngersZhuuuu and others added 3 commits December 14, 2021 10:44

Update docs/sql-distributed-sql-engine-spark-sql-cli.md

45ec6a0

Co-authored-by: Wenchen Fan <[email protected]>

Merge branch 'master' into SPARK-37558

403d966

Update sql-distributed-sql-engine-spark-sql-cli.md

d9f150c

Update sql-distributed-sql-engine-spark-sql-cli.md

453f260

cloud-fan reviewed Dec 14, 2021

View reviewed changes

docs/sql-distributed-sql-engine-spark-sql-cli.md Outdated Show resolved Hide resolved

cloud-fan reviewed Dec 14, 2021

View reviewed changes

cloud-fan approved these changes Dec 14, 2021

View reviewed changes

AngersZhuuuu and others added 2 commits December 15, 2021 14:31

Update docs/sql-distributed-sql-engine-spark-sql-cli.md

d4f6778

Co-authored-by: Wenchen Fan <[email protected]>

Update sql-distributed-sql-engine-spark-sql-cli.md

29f3b45

Update sql-distributed-sql-engine-spark-sql-cli.md

2b6f130

Update sql-distributed-sql-engine-spark-sql-cli.md

147d8b0

cloud-fan closed this in 2ffa2be Dec 15, 2021

[SPARK-37558][DOC] Improve spark sql cli document #34821

[SPARK-37558][DOC] Improve spark sql cli document #34821

Uh oh!

Conversation

AngersZhuuuu commented Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

AngersZhuuuu commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

SparkQA commented Dec 7, 2021

Uh oh!

AngersZhuuuu commented Dec 8, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

AngersZhuuuu commented Dec 6, 2021 •

edited

Loading